Measuring Researcher-Production in Information Systems

While many studies have assessed IS researcher-production, most have focused on either ranking IS journals or assessing prolific researchers using a restricted time frame and a small “basket” of journals (i.e., those journals selected for sampling). We found no research that has assessed the IS specificity of journals (i.e., the suitability of journals for publishing IS research) nor any that evaluated IS researcher-production measures. Based on a coding of over 26,000 articles and more than 1,900 authors, this study attempts such an evaluation by (1) determining the rate of publication of IS researchers in 58 journals perceived by at least one IS institution as IS specific, (2) profiling prolific and typical IS researchers using descriptive statistics, (3) evaluating the convergent validity of various researcher-production measures, (4) assessing the reliability of these researcher-production measures by varying baskets of Journal of the Association for Information Systems (Volume 3, 2002) 145-215 146 Measuring Researcher-Production in Information Systems by C. Chua, L. Cao, K. Cousins, & D. W. Straub journals and time periods, and (5) comparing the sensitivity of measures across prolific and typical researchers. The study demonstrates that many journals perceived to be of high quality by IS researchers are not specifically targeted to information systems. Changing the evaluation procedure has a significant impact on measures of typical and prolific IS researchers. For typical IS researchers, measures of production are strongly convergent and are not sensitive to changes in journal baskets. However, for prolific researchers, measures of production are not convergent and highly sensitive to changes in journal baskets. The evaluation of both prolific and typical IS researchers is also highly sensitive to temporal effects. The differences in convergent validity and reliability demonstrate that prolific researchers are more sensitive to minor variations in the assessment procedure. Based on the empirical findings, the study closes with recommendations both for the evaluation of researcher-production and for developing institutional target journal lists, i.e., lists of journals viewed favorably by an institution.